Super resolution image processing generally refers to techniques that enhance the resolution of an image or a video. Super-resolution in image and video technology enables an increase in the resolution of a low resolution image or a low resolution video to, respectively, a high resolution image or a high resolution video. By way of example, a television may use an image up sampling technique to convert a standard definition video sequence to a high definition video sequence. Similarly, an image or video taken with a low resolution imaging device, such as a mobile phone, may be converted to a high resolution image or video, respectively.
Many super resolution techniques use two processing stages. The first stage includes multiple low-resolution images being registered with an image processing system, where one of the low-resolution images is selected as a primary image, and horizontal and vertical displacements of the rest of the low-resolution images (i.e., secondary images) are computed with respect to the primary image. The second stage includes the registered images being combined together to create a super resolution image using the displacement information and aliasing information present in the low resolution images. This technique may be extended to a series of images of a video sequence for a video. Unfortunately, there are limits to the effectiveness of super resolution techniques following this approach. Furthermore; the resulting video sequence tends to have visual temporal jitter when the super resolution process is applied to subsequent images in the video sequence in an independent manner, even if the image content is substantially stationary.
A system to accurately determine a high resolution frame from a sequence of low resolution frames becomes more complex when the imaged scene in the input sequence of low resolution frames changes, as is often the case with real-world recorded video sequences. Additional difficulties arise when objects in the imaged scene move in a non-planar manner, when objects change shape in a non-rigid manner, or when multiple objects move independently of each other: Such complex scene motion often leads to complex changes in the captured image sequence that complicate the above-mentioned super resolution techniques. A practical example of video with such complex changes are video sequences of human faces. Furthermore, factors related to the recording process itself may contribute to image degradation, and variations in the processing of the low resolution frames.
The foregoing and other objectives, features, and advantages of the invention may be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.